570 research outputs found
Smoothed Gradients for Stochastic Variational Inference
Stochastic variational inference (SVI) lets us scale up Bayesian computation
to massive data. It uses stochastic optimization to fit a variational
distribution, following easy-to-compute noisy natural gradients. As with most
traditional stochastic optimization methods, SVI takes precautions to use
unbiased stochastic gradients whose expectations are equal to the true
gradients. In this paper, we explore the idea of following biased stochastic
gradients in SVI. Our method replaces the natural gradient with a similarly
constructed vector that uses a fixed-window moving average of some of its
previous terms. We will demonstrate the many advantages of this technique.
First, its computational cost is the same as for SVI and storage requirements
only multiply by a constant factor. Second, it enjoys significant variance
reduction over the unbiased estimates, smaller bias than averaged gradients,
and leads to smaller mean-squared error against the full gradient. We test our
method on latent Dirichlet allocation with three large corpora.Comment: Appears in Neural Information Processing Systems, 201
Sparse Stochastic Inference for Latent Dirichlet allocation
We present a hybrid algorithm for Bayesian topic models that combines the
efficiency of sparse Gibbs sampling with the scalability of online stochastic
inference. We used our algorithm to analyze a corpus of 1.2 million books (33
billion words) with thousands of topics. Our approach reduces the bias of
variational inference and generalizes to many Bayesian hidden-variable models.Comment: Appears in Proceedings of the 29th International Conference on
Machine Learning (ICML 2012
Hierarchical relational models for document networks
We develop the relational topic model (RTM), a hierarchical model of both
network structure and node attributes. We focus on document networks, where the
attributes of each document are its words, that is, discrete observations taken
from a fixed vocabulary. For each pair of documents, the RTM models their link
as a binary random variable that is conditioned on their contents. The model
can be used to summarize a network of documents, predict links between them,
and predict words within them. We derive efficient inference and estimation
algorithms based on variational methods that take advantage of sparsity and
scale with the number of links. We evaluate the predictive performance of the
RTM for large networks of scientific abstracts, web documents, and
geographically tagged news.Comment: Published in at http://dx.doi.org/10.1214/09-AOAS309 the Annals of
Applied Statistics (http://www.imstat.org/aoas/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Variational Inference in Nonconjugate Models
Mean-field variational methods are widely used for approximate posterior
inference in many probabilistic models. In a typical application, mean-field
methods approximately compute the posterior with a coordinate-ascent
optimization algorithm. When the model is conditionally conjugate, the
coordinate updates are easily derived and in closed form. However, many models
of interest---like the correlated topic model and Bayesian logistic
regression---are nonconjuate. In these models, mean-field methods cannot be
directly applied and practitioners have had to develop variational algorithms
on a case-by-case basis. In this paper, we develop two generic methods for
nonconjugate models, Laplace variational inference and delta method variational
inference. Our methods have several advantages: they allow for easily derived
variational algorithms with a wide class of nonconjugate models; they extend
and unify some of the existing algorithms that have been derived for specific
models; and they work well on real-world datasets. We studied our methods on
the correlated topic model, Bayesian logistic regression, and hierarchical
Bayesian logistic regression
- …